The present invention relates to a dynamically stable associative learning neural network system involving neuron circuits and networks and, more particularly, to a neuron circuit employing a novel learning rule which enables associative learning, including correlations and anti-correlations, with decreased computational time and complexity.
Efforts have been made to use neural networks to emulate human-like performance in the areas of pattern recognition including pattern-recognition applications in speech or image recognition, classification systems, and adaptive control systems. The basic computational element of a neural network is the neuron circuit which typically has multiple input lines and a single output line. The output response of a neuron circuit is generally a nonlinear function of the sum of the signal amplitudes on its input lines, with an output response being triggered when the sum of the input signal amplitudes exceeds a threshold value. The output of a neuron circuit may be coupled to the input of more than one other neuron circuit.
A neural network is formed by interconnecting the neuron circuits through synapses, each of which has an associated weight for modifying any signal passing through it. The amplitude of a signal exiting a synapse is thus the product of the weight associated with that synapse and the amplitude of the signal entering the synapse. A synapse may be either excitatory, that is, its weight is positive because it contributes to production of a signal by the associated neuron circuit, or inhibitory, that is, its weight is negative.
The output end of each synapse terminates at an input line to a neuron circuit, with the other end connected to either the output line of another neuron circuit or to a primary input (i.e., the receptor) to the neural network. The primary outputs of a neural network are each derived from a single output line of one of the neuron circuits in the system. With such loose restrictions, a large number of differently configured neural networks can be formed by simply varying the synaptic connections between neuron circuits.
The two major classes of artificial neural networks developed are (1) single layer networks in which a set of neuronal elements are fully interconnected with each other and which function well as associators, and (2) multilayer perceptrons in which all interconnections are feed-forward connections between layers and which function well as pattern classifiers. For the networks in these two classes, the basic neuronal models used are variants of a concept in which (1) "synaptic" inputs to an element are summed and the element fires if a threshold is exceeded, and (2) the weight or strength of a synaptic junction is increased only if both the presynaptic and post-synaptic elements fire. Essentially, all of the currently popular neural network designs implicitly or explicitly use the element to be formed in the adjustment of the weights on its input. In most implementations, a nonlinear optimization or relaxation algorithm is used for setting weights. These algorithms are all computationally intensive, since for each set of inputs to be learned many iterations are required before convergence is achieved and the network has learned the input patterns. Depending on the details of the implementation, the computational complexity of these algorithms (reflected in the number of iterations needed to achieve convergence) is between O(N.sup.2) and O(N.sup.3) where N is the number of weights (connections) to be determined. Consequently, the total computational effort per connection increases as the number of connections is increased. A survey of neural networks can be found in R. P. Lippmann, "An Introduction to Neural Nets," IEEE ASSP Magazine, Pgs. 4-21, April, 1987.
Neural networks are taught by successive presentation of sets of signals to their primary inputs with each signal set derived from a pattern belonging to a class of patterns, all having some common features or characteristics. Each time a set of signals is presented to the primary inputs, the synaptic weights must be adapted in order for the neural network to learn from this input.
The primary disadvantages associated with prior art neural networks result from their methods of adapting synaptic weights when the neural networks are in the training mode. Essentially, conventional neural network designs implicitly or explicitly use the output response of a given neuron circuit in adjusting the weights of the synapses connected to its input lines (see, for example, U.S. Pat. No. 3,950,733 issued Apr. 13, 1976 to Cooper et al). The use of the output response requires nonlinear optimization or relaxation algorithms which are computationally intensive; for a relatively small network of neuron circuits, thousands or ten of thousands of iterations are typically required before convergence is achieved and the network is said to have learned a set of input patterns. As the number of neuron circuits increase, the computational time to achieve convergence increases at a polynomial rate.
Additionally, most neural networks require a noise free external teacher that provides the neural network with a desired set of output values chosen to represent the class of patterns being learned. Sets of signals representing the exemplary input patterns are successively applied to the primary inputs and propagate through the neural network to the output. The differences between the actual and desired output vales, determined by the external teacher, are calculated to arrive at an error signal that is employed, typically with a nonlinear algorithm, throughout the network to adjust synaptic weights. This process is repeated in a recurring manner and typically requires a large number of iterations to reduce errors appearing at the primary outputs to an acceptable level.
To overcome some of the limitations of the art, a neural network has been developed which has many advantageous properties including the ability to learn, store and recall associations among vast numbers of noisy patterns. This network is qualitatively different from sum-of-weighted input models proposed by others. Also, other image understanding/object recognition systems are not based on associative learning.